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Pre-K Mathematics 

in the classroom through teacher-guided, small-group activities \with 
concrete manipulatives. Take-home activities with materials that 
parallel the small-group classroom activities are designed to help 
parents support their children’s mathematical development at home. 



Program doscription^ Pre-K Mathematics is a supplemental curriculum designed to 

develop informal mathematical knowledge and skills in preschool 
children. Mathematical content is organized into seven units. 
Specific mathematical concepts and skills from each unit are taught 



Research 



One study of Pre-K Mathematics met the What Works Clearing- 
house (WWC) evidence standards and one study met the WWC 
evidence standards with reservations.^ The studies included 430 
preschool children at 39 Head Start and state-funded preschools 
in New York and California and examined intervention effects 
on children’s math outcomes. This report focuses on immediate 



posttest findings to determine the effectiveness of the inter- 
vention.^ The WWC considers the extent of evidence for Pre-K 
Mathematics to be moderate to large for mathematics achieve- 
ment. No studies that met WWC evidence standards with or 
without reservations addressed oral language, print knowledge, 
phonological processing, early reading/writing, or cognition. 



Effectiveness Pre-K Mathematics combined with DLM Eariy Chiidhood Express software (hereafter referred to as DLM Express) was found to have 
positive effects on mathematics achievement. 



Print Phonological Early reading/ 





Oral language 


knowledge 


processing 


writing 


Cognition 


Math 


Rating of effectiveness 


na 


na 


na 


na 


na 


Positive effects 


Improvement index'^ 


na 


na 


na 


na 


na 


Average: -f22 
percentile points 
Range: -f22 to -f23 
percentile points 

na = not applicable 



1. The descriptive information for this program was obtained from pubiicly avaiiabie sources: the program’s web site (http://www.pearsonearlvlearnina. 
com/oroducts/curhculum/Dre k/index.html . downioaded March 15, 2007), the research iiterature (Clements & Sarama, 2006, June; Starkey & Kiein, 
2005; Starkey, Kiein, & Wakeley, 2004), and the program description provided by the author upon the WWC request. The WWC requests deveiopers to 
review the program description sections for accuracy from their perspective. Further verification of the accuracy of the descriptive information for this 
program is beyond the scope of this review. 

2. Both studies inciuded an additionai component with the Pre-K Mathematics intervention. The impact of the additional component— DLM Early Child- 
hood Express software— cannot be separated from the impact of the Pre-K Mathematics curriculum. However, because DLM Early Childhood Express 
software can be used with Pre-K Mathematics, the WWC considers the information in this report to be useful for practitioners in search of a preschool 
mathematics curriculum. To be eligible for the WWC’s review, the Early Childhood Education (ECE) intervention had to be implemented in English in 
center-based settings with children aged three to five or in preschool. 



WWC Intervention Report Pre-K Mathematics 



July 30, 2007 







Absence of conflict 
of interest 



Additionai program 
information^ 



WWC Intervention Report 



Pre-K Mathematics was developed in part by Dr. Alice Klein, 
the Principal Investigator for the WWC elementary school math 
topic review, and she receives royalties from sales of Pre-K 
Mathematics. In addition. Dr. Klein co-authored two of the 



studies reviewed for this WWC intervention report. Dr. Klein was 
not involved in the coding, reconciliation, or discussions of the 
included studies. Additionally, she was not involved in writing or 
reviewing the ECE intervention report on Pre-K Mathematics. 



Developer and contact 

Developed by Alice Klein and Prentice Starkey with Alma 
Ramirez, Pre-K Mathematics is distributed by Pearson Scott 
Foresman division of Pearson Education as part of the Scott 
Foresman-Addison Wesley Mathematics (2004) series for grades 
Pre-K-6. Address: 1900 E. Lake Avenue, Glenview, Illinois 60025. 
Web: www.scottforesman.com/support/index.cfm . Telephone: 
(800) 552-2259. 

Scope of use 

Information is not available on the number or demographics of 
children or centers using Pre-K Mathematics with or without 
DLM Express. 

Teaching 

The Pre-K Mathematics curriculum includes activities orga- 
nized in seven units containing closely related mathematical 
content: (1) Counting and Number, (2) Understanding Arithmetic 
Operations (Fall Activities), (3) Spatial Sense and Geometry, (4) 
Patterns, (5) Understanding Arithmetic Operations (Spring Activi- 
ties), (6) Measurement and Data, and (7) Logical Reasoning. The 
program has both classroom activities with manipulatives and 
home activities with picture strips in English or Spanish. In the 
classroom, mathematical concepts and skills are taught through 
teacher-guided, small-group activities over the course of the 
school year. A new mathematics activity is introduced each week 
and presented to pre-Kindergarten children twice. Each small 
group contains four to six children and the small-group lessons 



last approximately 20 minutes. Home activities are sent home 
with the child every one to two weeks. Parents are encouraged 
to help their children engage in mathematical activities that are 
related to the concepts and skills they are learning at school. 
Materials for the classroom and home are provided in a core 
program package that includes a teacher’s curriculum book, 
activity aid masters, home activity masters, and assessment 
record sheets. The research evidence reviewed in this report 
addresses Pre-K Mathematics combined with DLM Express, 
which is supplemental DLM Eariy Chiidhood Express math 
software. The software reinforced classroom support in two 
mathematical domains: (1) geometric and spatial concepts and 
skills and (2) numerical concepts and skills. It can be purchased 
only as part of a bundle called the DLM Express Math Resource 
Package, which includes the DLM math software CD and the 
math resources guide booklet. The studies reviewed for this 
report used only the math software, and in the case of Starkey 
and Klein (2005) only a subset of the math software activities 
included on the CD were used. 

Cost 

The Pre-K Mathematics Core Program Package (Pre-K 
Teacher’s Edition, Activity Aid Masters, Home Activity Masters, 
and Assessment Record Sheets) costs $145 and the Spanish 
Home Activity Masters cost $16. Information on the cost of 
professional development is not available. The DLM Express 
Math Resource Package that includes the computer software 
costs $185.® 



3. The evidence presented in this report is based on avaiiable research. Findings and conciusions may change as new research becomes avaiiabie. 

Pre-K Mathematics is a part of the Preschooi Curricuium Evaiuation Research (PCER) Grants administered through the U.S. Department of Education’s 
institute of Education Sciences. The finai PCER reports were not reieased in time to be reviewed for this report. 

4. These numbers show the average and range of student-ievei improvement indices for aii findings across the study. 

5. The manipulatives noted in the teaching section are not avaiiabie for purchase from Pearson Scott Foresman. 
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Research 



Effectiveness 



WWC Intervention Report 



Three studies reviewed by the WWC examined the effects 
of Pre-K Mathematics in center-based settings. One study 
(Starkey & Klein, 2005) was a randomized controiied trial that 
met WWC evidence standards. Another study (Ciements & 
Sarama, 2006) was a randomized controiied triai that met WWC 
evidence standards with reservations due to within-cluster 
differential attrition between the intervention and comparison 
groups. The remaining study did not meet WWC evidence 
screens. 

Met WWC evidence standards 

Starkey and Klein (2005) included two cohorts of 564 three- 
and four-year-old children from low-income families attending 
Head Start and state-funded preschool programs in New York 
and California, but the WWC includes the data from the 278 
children in cohort one only.® More than half of the children were 
African-American. The authors compared math outcomes for 
children who participated in a Pre-K Mathematics combined 
with DLM Express intervention group to children who partici- 
pated in a business-as-usual comparison group. 



Met WWC evidence standards with reservations 

Clements and Sarama (2006) included 21 preschool teachers 
(152 children) from low-income families in New York State and 
compared math outcomes for children participating in a Pre-K 
Mathematics combined with DLM Express intervention group to 
a business-as-usual comparison group.^ 

Extent of Evidence 

The WWC categorizes the extent of evidence in each domain as 
small or moderate to large (see the What Works Clearinghouse 
Extent of Evidence Categorization Scheme) . The extent of 
evidence takes into account the number of studies and the 
total sample size across the studies that met WWC evidence 
standards with or without reservations.® 

The WWC considers the extent of evidence for Pre-K Math- 
ematics to be moderate to large for mathematics achievement. 
No studies that met WWC evidence standards with or without 
reservations addressed oral language, print knowledge, phono- 
logical processing, early reading/writing, or cognition. 



Findings 

The WWC review of interventions for early childhood education 
addresses children’s outcomes in six domains: oral language, 
print knowledge, phonological processing, early reading/writing, 
cognition, and math.® Both studies addressed a single outcome 



(but used a different measure) in the math domain. The findings 
below present the authors’ and the WWC-calculated estimates 
of the size and statistical significance of the effects of Pre-K 
Mathematics on children’s mathematics performance. 



6. The study authors implemented a pure randomized controlled trial for cohort one (meets WWC evidence standards), whereas in cohort two the study 
authors replaced classrooms lost to attrition via random selection (meets WWC evidence standards with reservations). Therefore, the WWC bases the rat- 
ing of effectiveness on the data from cohort one only because it has the strongest research design. The data from cohort two are provided in Appendix A4. 

7. The study also included a Building Blocks for Math intervention group. The study authors labeled the Building Blocks for Math group as the “intervention 
group” and the Pre-K Mathematics group as the “comparison group;” however, the WWC considers Building Blocks for Math as a separate intervention 
(see the separate WWC Building Blocks for Math intervention report) . For the rating of effectiveness in this WWC intervention report, the WWC includes 
only the results comparing the Pre-K Mathematics group to the business-as-usual comparison group; however, results for the comparison between the 
curricula are included in a separate section of this report and Appendix A5. 

8. The Extent of Evidence Categorization was developed to tell readers how much evidence was used to determine the intervention rating, focusing on the 
number and sizes of studies. Additional factors associated with a related concept, external validity, such as the students’ demographics and the types 
of settings in which studies took place, are not taken into account for the categorization. 

9. The level of statistical significance was reported by the study authors or, where necessary, calculated by the WWC to correct for clustering within class- 
rooms or schools and for multiple comparisons. For an explanation about the clustering correction, see the WWC Tutorial on Mismatch . See Technical 
Details of WWC-Conducted Computations for the formulas the WWC used to calculate the statistical significance. In the case of Pre-K Mathematics, no 
corrections for clustering or multiple comparisons were needed. 
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Effectiveness (continued) 



The WWC found Pre-K 
Mathematics lo have 
positive effects on math 



Math. Starkey and Klein (2005) reported a statistically significant 
positive effect for Pre-K Mathematics combined \A/ith DLM 
Express for cohort one combined across states using the Child 
Math Assessment as an outcome measure. 

Clements and Sarama (2006) reported a statistically sig- 
nificant positive effect for Pre-K Mathematics combined with 
DLM Express using the Early Mathematics Assessment as an 
outcome measure. 



Rating of effectiveness 

The WWC rates the effects of an intervention in a given outcome 
domain as positive, potentially positive, mixed, no discernible 
effects, potentially negative, or negative. The rating of effective- 
ness takes into account four factors: the quality of the research 
design, the statistical significance of the findings,® the size of 
the difference between participants in the intervention and the 
comparison conditions, and the consistency in findings across 
studies (see the WWC Intervention Rating Scheme) . 



Improvement index 

The WWC computes an improvement index for each individual 
finding. In addition, within each outcome domain, the WWC 
computes an average improvement index for each study and 
an average improvement index across studies (see Technical 
Details of WWC-Conducted Computations) . The improvement 
index represents the difference between the percentile rank 
of the average student in the intervention condition versus 
the percentile rank of the average student in the comparison 
condition. Unlike the rating of effectiveness, the improvement 
index is based entirely on the size of the effect, regardless of 
the statistical significance of the effect, the study design, or the 
analyses. The improvement index can take on values between 
-50 and +50, with positive numbers denoting results favorable to 
the intervention group. 

The improvement index for math is +22 percentile points 
across the two studies, with a range of +22 to +23 percentile 
points across findings. 

Findings for comparisons between Pre-K Mathematics and 
Building Blocks for Math 

The analysis for the comparison described below was included 
in the Clements and Sarama (2006) study, but the findings do 
not contribute to the overall rating of effectiveness because the 
WWC included the comparison of Pre-K Mathematics to the 
business-as-usual comparison group in the rating for the same 
study, which provides a clearer sense of Pre-K Mathematics’ 



effects. However, the WWC believes that the findings from 
this comparison provide useful information to practitioners 
who may be interested in comparing the effects of different 
curricula. The WWC reports the findings for comparisons of 
Pre-K Mathematics and Buiiding Blocks for Math here and in 
Appendix A5. 

Math. Clements and Sarama (2006) analyzed group differ- 
ences between the Pre-K Mathematics combined with DLM 
Express group and the Building Blocks for Math group for one 
math outcome measure (Early Mathematics Assessment). The 
difference between groups was statistically significant and 
favored children in the Building Blocks for Math group. The 
improvement index for math is -19 percentile points (Pre-K 
Mathematics is the intervention group and Building Blocks for 
Math is the comparison group) for the single finding in the study. 

Summary 

The WWC reviewed three studies on Pre-K Mathematics. One 
study met WWC evidence standards and another study met 
WWC evidence standards with reservations; the remaining study 
did not meet WWC evidence screens. Based on these two stud- 
ies, the WWC found positive effects on math. Additional findings 
that were not considered for the rating of effectiveness indicated 
that Pre-K Mathematics may not have as large an impact on 
children’s math outcomes as another skills-focused preschool 
mathematics intervention. The evidence presented in this report 
may change as new research emerges. 
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For more information about specific studies and WWC caicuiations, piease see the WWC Pre-fC Mathematics 
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10. Lack of evidence for baseline equivalence: the study, which used a quasi-experimental design, did not establish that the comparison group was equiva- 
lent to the intervention group at baseline. 
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Appendix 



Appendix A1.1 Study characteristics: Starkey & Kiein, 2005 (randomized controiied triai) 



Characteristic 


Description 


Study citation^ 


Starkey, R, & Klein, A. (2005). A longitudinal study of the effects of a pre-kindergarten mathematics curriculum on low-income children’s mathematical knowledge (From PCER 
2002: Grantee Annual Progress Report (2005), lES Grant No. R305J020026). Berkeley: University of California. 


Participants 


The study was impiemented with two cohorts of children from two states across a two-year time period. This WWC review focuses on cohort one combined across states.^ 
During the first year of implementation, 20 Head Start and 20 state-funded preschool classrooms were randomly assigned to the Pre-K Mathematics intervention or business- 
as-usual comparison groups within program type. The classroom assignment to conditions was maintained for the second year of implementation. Eour intervention and two 
comparison classrooms from the first year of implementation were unable to participate and were replaced by randomly selecting classrooms from a list of volunteers in the 
second year.^ The study began with 316 low-income children in cohort one combined across states. During year one of the study, 38 children left resulting in a final cohort 
one sample of 278 children. The mean age of the children in cohort one was 4.4 years. Eifty-three percent of the children in cohort one were African-American; 22% were 
Hispanic; 22% were Caucasian; 4% were Asian-American; and 4% were interracial or another ethnicity. Eorty-eight percent of the children were female. 


Setting 


The study took place in New York and California in 40 Head Start and state-funded classrooms from 37 schools in cohort one and 40 Head Start and state-funded classrooms 
from 36 schools in cohort two. 


Intervention^ 


Teachers implemented the Pre-K Mathematics cumculum with small groups of 4-6 children in twice weekly 20-minute sessions for one school year. Each week involved the 
introduction of a new math activity. The sessions included activities in seven units: counting and number; understanding arithmetic operations (fall and spring units); spatial 
sense and geometry; patterns; measurement and data; and, logical reasoning. In addition, teachers used two other instructional activities: computer activities (DLM Express 
software) and mathematics learning centers, which included materials from the small-group activities and additional mathematics materials from the classroom. Home activity 
materials parallel to the classroom activities were sent home every one to two weeks for parents to use with their children. Teachers tracked children’s progress using a Math 
Mastery Eorm, and treatment fidelity data were collected using the Eidelity of Implementation Record Sheet. 


Comparison 


Children in the business-as-usual comparison group participated in the curriculum used in their schools (Creative Curriculum, High/Scope, Montessori, or specialized literacy 
curricula and curricula developed by local teachers and school districts). 


Primary outcomes 
and measurement 


The primary outcome domain assessed was math and it was measured with the researcher-developed Child Math Assessment (see Appendix A2 for a more detailed descrip- 
tion of the outcome measure). 


Teacher training 


During the first year of implementation teachers received one four-day workshop at the beginning of the school year to learn how to implement units one to three of the 
curriculum and another four-day workshop at mid-year to learn how to implement units four to seven of the curriculum. In addition, teachers were provided on-site training and 
implementation fidelity checks twice each month. In the second year of implementation, teachers received two-day refresher workshops at the beginning of the school year 
and at mid-year, and received on-site training and fidelity checks twice a month. 



1. The study authors submitted an executive summary to the WWC and provided additionai detaiis about participants, sampie sizes, study design, intervention impiementation, and the data 
necessary to caicuiate effect sizes upon WWC request. 

2. The study authors impiemented a pure randomized controiied trial for cohort one (meets WWC evidence standards), whereas in cohort two the study authors replaced classrooms lost to 
attrition via random selection (meets WWC evidence standards with reservations). Therefore, the WWC bases the rating of effectiveness on the data from cohort one only because it has the 
strongest research design. The data from cohort two are provided in Appendix A4. 

3. The impact of the DLM Express software cannot be separated from the impact of the Pre-K Mathematics curriculum. 
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Appendix A1.2 Study characteristics: Ciements & Sarama, 2006 (randomized controiied triai with attrition prohiems)^ 



Characteristic 


Description 


Study citation 


Clements, D. H., & Sarama, J. (2006, June). Scaling up the implementation of a pre-Kindergarten mathematics curriculum: The Building Blocks curriculum. Paper presented at 
the Institute of Education Sciences Research Conference, Washington, D.C. 


Participants 


Teachers were randomly assigned to conditions In two separate steps. Twenty-four teachers from preschool programs serving low-income children were randomly assigned to 
two intervention groups (Pre-K Mathematics or Building Blocks forMathf or a business-as-usual comparison group. Twelve teachers from preschool programs serving mixed- 
income children were randomly assigned to the Building Blocks for Math group or the business-as-usual comparison group. Consequently there were a total of eight teachers 
in the Pre-K Mathematics group and 14 teachers in the business-as-usual comparison group. Eight preschool-age children were randomly selected from each classroom for 
assessment (N=176).^ After attrition, the final sample included 21 teachers and 152 children (seven teachers and 51 children in the Pre-K Mathematics group; 14 teachers 
and 101 children in the business-as-usual comparison group). 


Setting 


The study was conducted in Head Start and state-funded preschool programs in New York State. 


Intervention 


Children in the intervention condition used the Pre-/(/Waf/jemat/cs curriculum, which was designed to develop informal mathematical knowledge and skills. The curriculum 
was implemented primarily during 15- to 20-minute small-group activities at least twice a week and 10- to 15-minute whole-class math activities once a week. In addition, 
these classrooms used DLM Express software for 5 to 10 minutes twice a week. Weekly letters were sent to parents that included math activities similar to those children were 
learning at school. The intervention lasted for 26 weeks and the teachers maintained their daily activities and schedule while inserting mathematics activities at appropriate 
times during the day.'* 


Comparison 


Children in the business-as-usual comparison group participated in their regular daily activities and schedule, with emphasis on small groups and computer activities. These 
included city-wide math activities. Creative Curriculum, Montessori math activities, or “home-grown” math materials based on state standards. 


Primary outcomes 
and measurement 


The primary outcome domain assessed was math and it was measured with the Early Mathematics Assessment (see Appendix A2 for a more detailed description of the 
outcome measure). The study authors also assessed implementation fidelity with the Eidelity of Implementation measure and the quality of the mathematics environment using 
the Classroom Observation of Early Mathematics Environment and Teaching. This WWC review does not include the results from these observations.^ 


Teacher training 


Professional development activities for teachers in the Pre-K Mathematics group consisted of four days of training, a monthly two-hour class, and monthly in-class coaching by 
project staff. Teacher training covered a number of topics such as supporting mathematical development in the classroom, recognizing and supporting mathematics through- 
out the day, setting up mathematics learning centers, teaching with computers, small-group activities, and supporting mathematical development in the home. 



1. The study was downgraded by the WWC due to within-cluster differential attrition of 11% between the Pre-K Mathematics group and the business-as-usual comparison group. Additionally, the 
two groups differed at pretest by more than half of a standard deviation. 

2. The study also included a Building Blocks for Math intervention group. The study authors labeled the Building Blocks for Math group as the “intervention group” and the Pre-K Mathematics 
group as the “comparison group”; however, the WWC considers Building Blocks for Math as a separate intervention (see the separate WWC Building Blocks for Math intervention reporti . For the 
rating of effectiveness in this WWC intervention report, the WWC includes only the results comparing the Pre-K Mathematics group to the business-as-usual comparison group; however, results 
for the comparison between the curricula are included in a separate section of this report and Appendix A5. 

3. The remaining 14 teachers were assigned to the Building Blocks for Math group, which is not the main focus of this WWC intervention report. 

4. The impact of the DLM Express software cannot be separated from the impact of the Pre-K Mathematics curriculum. Children in the Building Blocks for Math intervention group participated in 
10- to 15-minute small-group (4-6 children) math activities weekly. These children also participated in 5- to 15-minute whole-group math activities four times a week and 5- to 10-minute com- 
puter activities twice a week. Related family activities were sent home weekly. The intervention lasted for 26 weeks, and intervention teachers maintained their daily activities and schedule while 
inserting mathematics activities at appropriate times during the day. 

5. For further details about the outcomes included in the early childhood education topic review, please see the Early Childhood Education Protocol . 
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Appendix A2 Outcome measures in the math domain 



Outcome measure 


Description 


Child Math Assessment 


A researcher-developed measure designed to assess young children’s early mathematical knowledge in the areas of number, arithmetic, space and geometry, measurement 
and pattern knowledge (as cited in Starkey & Kiein, 2005).'' 


Early Mathematics 
Assessment 


A researcher-developed measure that uses two individual child interviews to assess young children’s mathematical knowledge and skills in the areas of number, geometry, 
measurement, and patterning (as cited in Clements & Sarama, 2006). 


1 . The Child Math Assessment was developed by the researchers, who are also the program developers, and this measure was developed for the purposes of this research project. The measure 
was confirmed to have sufficient face validity by the WWC ECE Principal Investigators and a psychometric study to establish its measurement properties by the study authors as a part of the 



Institute of Education Sciences funded Interagency Education Research Initiative (lERI) Scale-Up project. 
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Appendix A3 Summary of study findings inciuded in the rating for the math domain^ 









Authors’ findings from the study 














Mean outcome 
(standard deviation^) 




WWC calculations 




Outcome measure 


Study 

sample 


Sample size 
(classrooms/ 
children)^ 


Pre-K Math Comparison 

group'* group'* 


Mean difference^ 
{Pre-K Math - 
comparison) 


Statistical 
significance^ 
Effect size® (at a = 0.05) 


Improvement 

index® 



Child Math Assessment 3-4 year olds 


40/278 


Starkey & Klein, 2005 (randomized controlled trial)® 

0.55 0.47 0.08 

(0.13) (0.14) 


0.58 


Statistically 

significant 


-f22 


Average*® for math (Starkey & Klein, 2005) 








0.58 


Statistically 

significant 


-f22 




Clements & Sarama, 2006 (randomized controlled trial with attrition problems)** 








Early Mathematics Assessment Preschool children 


21/152 


58.01 


53.22 4.79 


0.61 


Statistically 


-f23 






(6.53) 


(8.38) 




significant 




Average*® for math (Clements & Sarama, 2006) 








0.61 


Statistically 


-f23 












significant 




Domain average*® for math across all studies 








0.60 


na 


-f22 



na = not applicable 

1 . This appendix reports findings considered for fhe effecfiveness rafing and the average improvement indices. For Starkey and Kiein (2005), the findings for cohorf one are combined across stafe; findings for cohort two are reported in 
Appendix A4. For Ciements & Sarama (2006), findings from the head-to-head comparison of Pre-K Mathematics to Building Blocks for Math are not inciuded in these ratings, but are reported in Appendix A5. 

2. The standard deviation across aii students in each group shows how dispersed the participants’ outcomes are: a smaiier standard deviation on a given measure wouid indicate that participants had more simiiar outcomes. For Starkey 
and Kiein (2005), the standard deviations were provided by the study authors upon WWC request. 

3. For Ciements and Sarama (2006), the ciuster-unit is the teacher and the chiid-ievei sampie size was provided by the study authors upon WWC request. For Starkey and Kiein (2005), the ciassroom and chiid-ievei sampie sizes were 
provided by the study authors upon WWC request. 

4. For Ciements and Sarama (2006), the intervention group mean (58.01) equais the comparison group mean (53.22) pius the program coefficient from the author-conducted FiLM anaiysis (4.79). For Starkey and Kiein (2005), the posttest 
means are covariate-adjusted means provided by the study authors upon WWC request. 

5. Positive differences and effect sizes favor fhe intervenfion group; negafive differences and effecf sizes favor fhe comparison group. For Ciemenfs and Sarama (2006), fhe mean difference was the program coefficient from fhe aufhor- 
conducfed HLM anaiysis. 

6. For an expianafion of the effect size caicuiation, see Technicai Detaiis of WWC-Conducted Computations . 

7. Statisticai significance is the probabiiity that the difference befween groups is a resuif of chance rafher fhan a reai difference between the groups. 

8. The improvement index represents the difference between the percentiie rank of fhe average sfudenf in the intervention condition versus the percentiie rank of fhe average sfudent in the comparison condition. The improvement index 
can take on vaiues between -50 and -i-50, with positive numbers denoting resuits favorabie to the intervention group. 

9. The ievei of stafisticai significance was reported by the study authors or, where necessary, caicuiated by the WWC to correct for ciusfering within ciassrooms or schoois and for muifipie comparisons. For an expianafion about the cius- 
tering correction, see the WWC Tutoriai on Mismatch . See Technicai Detaiis of WWC-Con ductec‘ Co mputafions for the formuias the WWC used to caicuiate statisticai significance, in fhe case of Sfarkey and Kiein (2005), no correcfions 
for ciustering or muitipie comparisons were needed because the study authors used HLM to anaiyze their data and accounted for ciusfering of chiidren in ciassrooms. 

10. The WWC-computed domain average effecf sizes for each study and for the domain across studies are simpie averages rounded to two decimai piaces. The average improvement indices are caicuiated from fhe average effecf sizes. 

11. in the case of Ciemenfs and Sarama (2006), no correcfions for ciusfering or muifipie comparisons were needed because the study authors used HLM to anaiyze their data and accounted for ciustering of chiidren in ciassrooms. 
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Appendix A4 Summary of findings from cohort two for the math domain^ 









Authors’ findings from the study 














Mean outcome 
(standard deviation^) 




WWC calculations 




Outcome measure 


Study 

sample 


Sample size 
(classrooms/ 
children)^ 


Pre-K Math Comparison 

group'* group'* 


Mean difference^ 
{Pre-K Math - 
comparison) 


Statistical 
significance^ 
Effect size® (at a = 0.05) 


Improvement 

index® 







Starkey & Klein, 2005 (randomized controlled trial; Cohort 2)® 








Child Math Assessment 


3-4 year olds 


40/286 0.58 0.47 0.11 

(0.16) (0.15) 


0.70 


Statistically 

significant 


-f26 



1 . This appendix presents findings for cohort two combined across state for measures that faii in the math domain. Combined scores across state for cohort one were used for rating purposes and are presented in Appendix A3. 

2. The standard deviation across aii students in each group shows how dispersed the participants’ outcomes are: a smaiier standard deviation on a given measure wouid indicate that participants had more simiiar outcomes. The posttest 
standard deviations were provided by the study authors upon WWC request. 

3. The sampie sizes were provided by the study authors upon WWC request. 

4. The posttest means are covariate-adjusted means provided by the study authors upon WWC request. 

5. Positive differences and effect sizes favor the intervention group; negative differences and effect sizes favor the comparison group. 

6. For an expianation of the effect size caicuiation, see Technicai Detaiis of WWC-Conducted Computations . 

7. Statisticai significance is the probabiiity that the difference between groups is a resuit of chance rather than a reai difference between the groups. 

8. The improvement index represents the difference between the percentiie rank of the average student in the intervention condition versus the percentiie rank of the average student in the comparison condition. The improvement index 
can take on vaiues between -50 and -t-50, with positive numbers denoting resuits favorabie to the intervention group. 

9. The ievei of statisticai significance was reported by the study authors or, where necessary, caicuiated by the WWC to correct for ciustering within ciassrooms or schoois (corrections for muitipie comparisons were not done for findings 
not inciuded in the overaii intervention rating). For an expianation about the ciustering correction, see the WWC Tutoriai on Mismatch . See Technicai Detaiis of WWC-Con ducted Co mputations for the formuias the WWC used to caicuiate 
statisticai significance, in the case of Starkey and Kiein (2005), no correction for ciustering was needed because the study authors used HLM to anaiyze their data and accounted for ciustering of chiidren in ciassrooms. 
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Appendix A5 Summary of findings for comparisons between Pre-K Mathematics and Building Blocks for Math for the math domain^ 









Authors’ findings from the study 














Mean outcome 
(standard deviation^) 




WWC calculations 




Outcome measure 


Study 

sample 


Sample size 
(teachers/ 
children)^ 


Building Blocks 
Pre-K Math for Math 

groups group’’ 


Mean difference^ 
(Pre-K Math - 
Building Blocks 
for Math) 


Statistical 
significance^ 
Effect size® (at a = 0.05) 


Improvement 

index® 





Clements & Sarama, 2006 (randomized controlled trial with attrition problems)® 






Early Mathematics Assessment Preschool children 


21/152 55.84 59.39 -3.55 -0.49 

(6.53) (7.46) 


Statistically 

significant 


-19 


Domain average’® for math 


-0.49 


Statistically 

significant 


-19 



1 . This appendix presents findings for the head-to-head comparison of Pre-K Mathematics and Building Blocks for Math for a measure that faiis in the math domain. Comparisons of Pre-K Mathematics and the business-as-usuai compari- 
son group were used for rating purposes and are presented in Appendix A3. 

2. The standard deviation across aii students in each group shows how dispersed the participants’ outcomes are: a smaiier standard deviation on a given measure wouid indicate that participants had more simiiar outcomes. The standard 
deviations were provided by the study authors upon WWC request. 

3. The chiid-ievei sampie size was provided by the study authors upon WWC request. 

4. The Pre-K Mathematics group mean (55.84) equais the Building Blocks for Math group mean (59.39) pius the program coefficient from the author-conducted HLM anaiysis (-3.55). 

5. Positive differences and effect sizes favor the Pre-K Mathematics group; negative differences and effect sizes favor the Building Blocks for Math group. The mean difference was the program coefficient from the author-conducted HLM 

anaiysis. 

6. For an expianation of the effect size caicuiation, see Technicai Detaiis of WWC-Conducted Computations . 

7. Statisticai significance is the probabiiity that the difference between groups is a resuit of chance rather than a reai difference between the groups. 

8. The improvement index represents the difference between the percentiie rank of the average student in the Pre-K Mathematics cond\tion versus the percentiie rank of the average student in the Building Blocks for Math condition. The 
improvement index can take on vaiues between -50 and -t-50, with positive numbers denoting resuits favorabie to the Pre-K Mathematics group. 

9. The ievei of statisticai significance was reported by the study authors or, where necessary, caicuiated by the WWC to correct for ciustering within ciassrooms or schoois and for muitipie comparisons. For an expianation about the 
ciustering correction, see the WWC Tutoriai on Mismatch . See Technicai Detaiis of WWC-Conducted Computations for the formuias the WWC used to caicuiate statisticai significance, in the case of Ciements and Sarama (2006), no cor- 
rections for ciustering or muitipie comparisons were needed because the study authors used HLM to anaiyze their data and accounted for ciustering of chiidren in ciassrooms. 

10. This row provides the study average, which in this instance, is aiso the domain average. The WWC-computed domain average effect size is a simpie average rounded to two decimai piaces. The domain improvement index is caicuiated 
from the average effect size. 
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Appendix A6 Pre-K Mathematics rating for the math domain 



The WWC rates an intervention’s effects in a given outcome domain as positive, potentiaily positive, mixed, no discernibie effects, potentialiy negative, or negative.'' 

For the outcome domain of math, the WWC rated Pre-K Mathematics as having positive effects. The remaining ratings (potentiaiiy positive effects, mixed effects, no 
discernible effects, potentiaily negative effects, and negative effects) were not considered, as Pre-K Mathematics was assigned the highest appiicabie rating. 



Rating received 

Positive effects: Strong evidence of a positive effect with no overriding contrary evidence. 

• Criterion 1: Two or more studies showing statisticaiiy significant positive effects, at least one of which met WWC evidence standards for a strong design. 

Met. Both studies showed statisticaiiy significant positive effects and one of the studies met WWC evidence standards for a strong design. 

AND 

• Criterion 2: No studies showing statisticaiiy significant or substantively important negative effects. 

Met. The studies did not show statisticaiiy significant or substantiveiy important negative effects. 

1. For rating purposes, the WWC considers the statistical significance of individual outcomes and the domain-level effect. The WWC also considers the size of the domain-level effect for ratings of 
potentially positive or potentially negative effects. See the WWC Intervention Rating Scheme for a complete description. 
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Appendix A7 Extent of evidence by domain 



Outcome domain 


Number of studies 


Schoois 


Sample size 

Classrooms/children 


Extent of evidence^ 


Oral language 


0 


0 




0 


na 


Print knowledge 


0 


0 




0 


na 


Phonological processing 


0 


0 




0 


na 


Early reading/writing 


0 


0 




0 


na 


Cognition 


0 


0 




0 


na 


Math 


2 


37+ 




61/430 


Moderate to large 



na = not applicable/not studied 

1 . A rating of “moderate to large” requires at least two studies and two schools across studies in one domain, and a total sample size across studies of at least 350 students or 14 classrooms. 
Otherwise, the rating is “small.” 
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